Various commercial and open-source tools exist to assist with the automation of test cases for software testing. The automation of test cases requires developing a comprehensive approach to test all of the functional areas that the software is deployed on. This may involve test cases invoked on duplicative frameworks to fully test software applications with existing tools. For example, one test case may be required to test a functional area such as a website user interface, whereas a separate test case may be required to automate CLI test case scenarios used in connection with the website. Likewise, if a user wants to implement testing of a Java application and a web service running components of this Java application, the user may build separate components to consume each of the Java application and the web service.
Once suitable test cases have been designed and selected, automated testing systems can automatically execute the test cases using the application under test and verify the performance of the application under test.
Typically, software components are continually updated by their authors to provide new functionality, fix bugs, ensure interoperability with new versions of other software or operating systems, etc. It is desirable to test updated software components before the revisions are released to users.
Such testing, referred to as regression testing, is designed to identify if any new software bugs, or regressions, have been introduced into an application after the application has been changed. Regression testing may be further designed to determine whether a change in one part of the application has affected other parts of the application, which is a common occurrence.
Regression testing typically involves re-running the original test cases that were performed when the application was initially developed and checking to see whether any new faults have been introduced or previously fixed faults have re-emerged. It may not be necessary to execute all of the test cases against the application under test during regression testing, as not all of the test cases may invoke the changed code. Thus, a large part of regression testing includes systematically selecting an appropriate set of tests needed to adequately cover a particular change. However, a regression test that only addresses changed portions of code may not adequately test the application.